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Abstract 


Over  the  last  few  decades,  services  acquisition  has  continued  to  increase  in 
scope  and  dollars  obligated.  Contracting  for  services  has  grown  in  relation  to 
systems  contracting  over  the  last  couple  of  decades  and  is  the  fastest  growing 
procurement  sector  for  the  DoD.  This  growth  in  dollars  obligated  has  attracted 
increased  political  attention  and  scrutiny  on  an  already  problematic  defense 
contracting  process.  The  DoD  has  responded  to  these  problems  by  improving 
services  acquisition  in  several  different  ways,  but  even  with  these  improvements, 
services  acquisition  still  has  problems  in  the  areas  of  procurement  planning,  source 
selection,  and  contract  administration.  This  research  continues  our  ongoing 
investigation  in  DoD  services  acquisition  by  exploring  the  determinants  of  contract 
success.  We  use  the  DoD  Contractor  Performance  Assessment  Reporting  System 
(CPARS)  as  a  proxy  for  contract  success  and  determine  if  there  are  any 
relationships  between  contract  variables  (type  of  service,  contract  dollar  value,  level 
of  competition,  contract  type)  and  contract  success  based  on  CPARS  ratings  (quality 
of  product/service,  schedule,  cost  control,  business  relations,  management  of  key 
personnel,  and  utilization  of  small  business).  Our  research  findings  revealed  that 
contract  dollar  value  and  level  of  competition  affected  the  success  of  a  service 
contract.  The  findings  also  revealed  that  the  failure  rate  in  CPARS  was  lower  than 
expected.  Finally,  we  saw  that  as  the  percentage  of  1 102  filled  billets  increased,  the 
contract  failure  rate  decreased.  We  also  observed  that  as  workload  dollars  per  filled 
billet  increased,  contractor  performance  ratings  also  increased,  and  thus  contract 
failure  ratings  decreased.  From  these  findings,  the  report  presents  a  discussion  of 
the  results  and  the  managerial  implications. 

Keywords:  Services  Acquisition,  Services  Contracts,  Success  of  Services 
Contracts 
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Services  Supply  Chain  in  the  Department  of 
Defense:  Drivers  of  Success  in  Services 

Acquisition 


Introduction 

The  service  sector  represents  the  largest  and  the  fastest-growing  segment  of 
the  economies  of  the  United  States  and  other  developed  countries.  This  growth  of 
services  in  the  overall  economy  is  also  mirrored  by  the  growth  of  services  acquisition 
in  the  Department  of  Defense  (DoD).  For  example,  DoD  obligations  on  contracts 
have  more  than  doubled  between  fiscal  years  2001  and  2008  to  over  $387  billion, 
with  over  $200  billion  spent  just  for  services  in  2008  (Government  Accountability 
Office  [GAO],  2009a).  However,  during  this  time  period,  the  acquisition  workforce 
declined  from  about  500,000  to  about  200,000  personnel  by  2006  (Gansler,  201 1 ,  p. 
237).  The  downsizing  of  the  defense  acquisition  workforce  has  reduced  the  qualified 
contracting  and  acquisition  workforce  necessary  to  manage  the  increased  service 
contract  workload  (GAO,  2002b,  2009b). 

According  to  the  GAO,  the  poor  management  of  service  contracts  has 
undermined  the  government’s  ability  to  obtain  a  good  value  for  the  money  spent  and 
has  contributed  to  the  GAO’s  decision  to  designate  management  of  services 
contracts  as  a  high-risk  area  for  the  DoD  (GAO,  2013).  In  fact,  as  stressed  in  a 
recent  memorandum  for  acquisition  professionals  from  the  Under  Secretary  of 
Defense  for  Acquisition,  Technology,  and  Logistics,  improving  the  efficiency  of 
acquisition  of  products  and  services  is  of  utmost  importance  to  the  DoD  (Under 
Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics  [USD(AT&L)], 
2010a).  More  specifically,  in  a  later  memorandum,  the  USD(AT&L)  has  focused  on 
“improving  tradecraft  in  services  acquisition”  by  strengthening  and  improving  the 
services  contracting  process  (USD[AT&L],  2010b,  p.  5). 

As  the  DoD’s  services  acquisition  continues  to  increase,  the  agency  must 
give  greater  attention  to  proper  acquisition  planning,  adequate  requirements 
definition,  sufficient  price  evaluation,  and  proper  contractor  oversight  (GAO,  2002b). 
In  some  ways,  the  issues  affecting  services  acquisition  are  similar  to  those  affecting 
the  acquisition  of  physical  supplies  and  weapon  systems.  However,  the  unique 
characteristics  of  services  and  the  increasing  importance  of  services  acquisition  offer 
a  significant  opportunity  for  conducting  research  in  the  management  of  services 
acquisition  in  the  Department  of  Defense. 
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Research  Methodology 

The  objective  of  this  research  is  to  identify  variables  in  the  services 
contracting  process  that  drive  the  success  of  services  acquisition.  Based  on  the 
analysis  of  our  data,  we  generalize  our  research  findings  and  provide 
recommendations  for  improving  the  Army’s  as  well  as  DoD’s  services  acquisition 
management.  Our  research  approach  includes  analyzing  contractor  past 
performance  data  obtained  from  the  DoD  Past  Performance  Information  Retrieval 
System  Report  Cards  (PPIRS-RC)  database.  The  PPIRS-RC  database  consists  of 
contractor  performance  assessment  reports  contained  in  the  DoD  Contractor 
Performance  Assessment  Reporting  System  (CPARS).  We  access  the  PPIRS-RC 
database  to  collect  contractor  performance  ratings  on  completed  services  contracts 
provided  by  CPARS  to  determine  whether  the  contracts  are  successful  or  not 
successful.  Statistical  analysis  is  used  to  draw  conclusions  on  whether  certain 
contracting  variables  affect  the  success  of  the  contract.  The  specific  contract 
variables  analyzed  are  type  of  service,  contract  dollar  value,  level  of  competition, 
and  contract  type.  The  following  are  the  specific  questions  answered  in  this 
research: 

1 .  Do  the  types  of  services  being  acquired  affect  the  success  of  a  service 
contract? 

2.  Do  the  contractual  amounts  affect  the  success  of  a  service  contract? 

3.  Does  the  level  of  competition  used  affect  the  success  of  a  service 
contract? 

4.  Does  the  contract  type  affect  the  success  of  a  services  contract? 

Literature  Review 

The  academic  research  in  the  management  of  services  acquisition  is  founded 
on  several  economic  and  management  theories  including  agency  theory 
(Eisenhardt,  1989),  transaction  cost  economics  (Williamson,  1979),  contractual 
theory  (Luo,  2002),  service  operations  and  supply  management  (Fitzsimmons  & 
Fitzsimmons,  2006),  and  stakeholder  theory  (Cleland,  1986;  El-Gohary,  Osman,  & 
El-Diraby,  2006;  Freeman,  1984).  For  the  purpose  of  this  research,  our  focus  is  on 
agency  theory  and  the  principal-agent  problem. 

Agency  Theory 

Agency  theory  is  reflected  in  a  contract  between  the  government  and  a 
contractor,  forming  a  principal-agent  relationship.  The  principal  (government) 
contracts  with  the  agent  (contractor)  to  perform  a  specific  level  of  effort,  such  as 
developing  or  manufacturing  a  product  or  providing  a  service.  In  this  relationship,  the 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


government’s  objectives  include  obtaining  the  product  or  service  at  the  right  quality, 
right  quantity,  right  source,  right  time,  and  right  price  (Lee  &  Dobler,  1971).  The 
principal,  in  this  case  the  federal  government,  also  has  the  additional  objective  of 
ensuring  the  product  or  service  is  procured  in  accordance  with  public  policy  and 
meets  the  needs  of  the  public  interest  (Cohen  &  Eimicke,  2008;  Snider  &  Rendon, 
2008).  Contractors,  on  the  other  hand,  pursue  the  objectives  of  earning  profit, 
ensuring  company  growth,  maintaining  or  increasing  market  share,  and  improving 
cash  flow,  just  to  name  a  few.  Because  of  the  different  and  conflicting  objectives 
between  the  principal  and  agent,  each  party  is  motivated  and  incentivized  to  behave 
in  a  certain  manner.  This  behavior  includes  either  withholding  or  sharing  information. 
In  principal-agent  relationships  that  involve  higher  levels  of  uncertainty,  which  result 
in  higher  risk  (such  as  developing  an  advanced  technology  weapon  system),  the 
information  available  to  the  government  and  contractor  is  typically  asymmetrical. 
Thus,  agency  theory  is  concerned  with  the  conflicting  goals  between  the  principal 
and  agent  in  obtaining  their  respective  objectives  and  is  focused  on  mechanisms 
related  to  obtaining  information  (for  example,  about  the  marketplace,  about  the 
supply  or  service,  or  about  the  contractor);  selecting  the  agent  (to  counter  the 
problem  of  adverse  selection);  and  monitoring  the  agent’s  performance  (to  counter 
the  effects  of  moral  hazard).  Thus,  decisions  about  how  contracts  are  planned  (for 
example,  competitive  or  sole  source),  structured  (fixed  price  or  cost  reimbursement, 
with  or  without  incentives),  awarded  (based  on  lowest  priced/technically  acceptable 
offer,  or  the  highest  technically  rated  offer),  and  administered  (centralized  or 
decentralized,  level  and  type  of  surveillance,  and  use  of  project  teams)  have  their 
basis  in  agency  theory  and  the  principal-agent  problem.  These  aspects  of  agency 
theory  are  directly  applicable  to  services  acquisition  management.  The  next  section 
discusses  our  past  research  in  this  area. 

Services  Supply  Chain  Management 

We  have  addressed  the  need  for  research  in  this  increasingly  important  area 
of  services  acquisition  by  undertaking  six  sponsored  research  projects  over  the  past 
six  years.  The  first  two  research  projects  (Apte,  Ferrer,  Lewis,  &  Rendon,  2006;  Apte 
&  Rendon,  2007)  were  exploratory  in  nature,  aimed  at  understanding  the  types  of 
services  being  acquired,  the  associated  rates  of  growth  in  services  acquisition,  and 
the  major  challenges  and  opportunities  present  in  the  service  supply  chain. 

The  next  two  research  projects  were  survey-based  empirical  studies  aimed  at 
developing  a  high-level  understanding  of  how  services  acquisition  is  currently  being 
managed  at  a  wide  range  of  Army,  Navy,  and  Air  Force  installations  (Apte,  Apte,  & 
Rendon,  2008,  2009).  The  analysis  of  survey  data  indicated  that  the  current  state  of 
services  acquisition  management  suffers  from  several  deficiencies,  including  deficit 
billet  and  manning  levels  (which  are  further  aggravated  by  insufficient  training  and 
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the  inexperience  of  acquisition  personnel)  and  the  lack  of  strong  project-team  and 
life-cycle  approaches.  Our  research  (Apte,  Apte,  &  Rendon,  2010)  also  analyzed 
and  compared  the  results  of  the  primary  data  collected  in  two  previous  empirical 
studies  involving  Army,  Navy,  and  Air  Force  contracting  organizations  so  as  to 
develop  a  more  thorough  and  comprehensive  understanding  of  how  services 
acquisition  is  being  managed  within  individual  military  departments. 

As  a  result  of  these  research  projects  dealing  with  the  service  supply  chain  in 
the  DoD,  we  developed  a  comprehensive,  high-level  understanding  of  services 
acquisition  in  the  DoD,  identified  several  specific  deficiencies,  and  proposed  a 
number  of  concrete  recommendations  for  performance  improvement. 

Based  on  the  foundation  of  the  previously  mentioned  management  theories, 
conclusions  of  the  GAO  and  DoDIG  reports  (Seifert  &  Ermoshkin,  2010),  and 
findings  of  our  own  sponsored  research  projects  on  the  topic,  we  believe  that  the 
success  of  services  acquisition  contracts  is  significantly  influenced  by  four  broadly 
defined  factors:  (1 )  the  type  and  quantity  of  services  being  outsourced  and  the 
associated  amount  of  acquisition-related  workload;  (2)  the  characteristics  of 
contracts  being  awarded;  (3)  the  capacity  available  to  carry  out  the  contracting, 
project  management,  and  surveillance  work;  and  (4)  various  management  practices 
such  as  use  of  project  team  or  life-cycle  approaches  and  so  forth.  A  conceptual 
model  indicating  the  interrelationship  among  these  factors  is  shown  in  Figure  1 . 


Figure  1.  Drivers  of  Acquisition  Practices  and  Success  of  Service 

Contracts 
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As  shown  in  the  conceptual  diagram  of  Figure  1 ,  the  contract  characteristics 
are  affected  by  the  type  of  service  being  acquired,  while  the  management  practices 
being  used  are  influenced  by  the  services  being  acquired,  the  contract 
characteristics,  and,  more  important,  the  capacity  available  to  perform  the 
acquisition  work.  The  success  of  services  contracts,  in  turn,  is  affected  by  the 
previously  mentioned  four  drivers.  Underlying  Figure  1  is  the  fundamental  question 
motivating  our  in-depth  research:  What  drives  the  success  of  services  contracts? 
This  fundamental  question  is,  of  course,  critically  important,  and  yet  it  is  also  not  one 
that  can  be  answered  easily  or  quickly.  We  believe  that,  generally,  in  the  case  of 
questions  related  to  complex  systems,  it  is  preferable  to  break  down  the  overall 
system  in  smaller  parts,  gain  an  understanding  of  the  functioning  of  each  part,  and 
then  put  all  the  pieces  together  to  better  understand  the  overall  system  and  answer 
the  fundamental  question. 

Based  on  our  conceptual  model,  we  sought  to  understand  how  the  success  of 
services  contracts  is  being  defined  and  measured  by  different  stakeholders.  On  the 
aggregate  level,  our  research  indicated  that,  when  defining  a  successful  service 
contract,  stakeholders  considered  outcomes  (in  the  order  of  performance,  cost,  and 
schedule)  slightly  more  important  than  processes.  Stakeholders  also  ranked 
outcome-related  factors  as  most  important.  Additionally,  on  the  aggregate,  our 
research  indicated  that,  when  measuring  a  successful  service  contract,  stakeholders 
considered  outcomes  (in  the  order  of  cost,  schedule,  and  performance)  more 
important  than  processes.  Stakeholders  also  ranked  outcome-related  factors  as 
most  important.  On  the  stakeholder  level,  our  research  indicated  that,  when  defining 
a  successful  service  contract,  PMs,  CORs,  and  COs  considered  outcomes  (in  the 
order  of  performance,  cost,  and  schedule)  slightly  more  important  than  processes. 
PMs,  CORs,  and  COs  also  ranked  outcome-related  factors  as  most  important.  On 
the  stakeholder  level,  our  research  indicated  that,  when  measuring  a  successful 
service  contract,  PMs,  CORs,  and  COs  considered  outcomes  (in  the  order  of 
performance,  schedule,  and  cost)  more  important  than  processes.  PMs,  CORs,  and 
COs  also  ranked  outcome-related  factors  as  most  important  (Apte  &  Rendon,  2013). 

Building  on  these  research  findings  concerning  how  stakeholders  define  and 
measure  the  success  of  services  contracts,  we  now  explore  the  question  of  what 
variables  in  the  services  contracting  process  drive  the  success  of  services 
acquisition.  As  previously  discussed,  our  research  on  defining  and  measuring 
services  contracting  success  found  that,  on  the  aggregate,  stakeholders  considered 
outcomes  slightly  more  important  than  processes  for  both  defining  and  measuring 
success.  Thus,  we  adopt  contract  outcomes,  specifically  the  outcomes  as  reflected 
in  the  contractor  performance  assessment  report,  as  a  proxy  for  contract  success. 
The  next  section  provides  a  brief  background  on  contractor  performance  information. 
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Contractor  Performance  Information 

Contractor  performance  information  is  information  regarding  a  contractor’s 
performance  under  previously  awarded  contracts.  Federal  procurement  policy 
requires  that  agencies  collect  contractor  performance  information  for  contracts  over 
$100,000  and  make  that  information  available  for  use  in  future  contract  award 
decisions  (Nash,  Schooner,  O’Brien-Debakey,  &  Edwards,  2007).  The  collection  of 
contractor  performance  information  occurs  during  the  contract  closeout  phase  using 
the  DoD  Contractor  Performance  Assessment  Reporting  System  (CPARS;  Rendon 
&  Snider,  2008). 

The  CPARS  assessment  data  reflect  the  contractor’s  performance  in  specific 
areas  including  quality,  schedule,  cost  control,  business  relations,  management  of 
key  personnel,  and  utilization  of  small  business.  The  Quality  rating  assesses  the 
contractor’s  qualitative  performance  and  compares  it  to  the  requirements  stated  in 
the  contract.  The  Schedule  rating  assesses  the  contractor’s  ability  to  meet 
schedules  outlined  in  the  contract  such  as  milestones,  task  orders,  delivery 
schedules,  and  administrative  requirements.  The  Cost  Control  rating  assesses  the 
contractor’s  ability  to  forecast,  manage,  and  control  the  costs  associated  with 
performing  their  services.  The  Business  Relations  rating  assesses  the  contractor’s 
ability  to  coordinate  their  business  activities  such  as  cooperative  behavior,  customer 
satisfaction,  management,  and  attitude  towards  customers.  The  Management  of  Key 
Personnel  rating  assesses  the  contactor’s  ability  to  maintain  qualified  individuals  in 
key  positions  as  outlined  in  the  contract.  The  Utilization  of  Small  Business  rating 
assesses  the  contractor’s  ability  to  integrate  small  businesses  in  the  execution  of  the 
contract  (Hart,  Stover,  &  Wilhite,  2013). 

Additionally,  the  CPARS  assessment  rates  the  contractor  in  these  areas 
using  the  rating  scales  Exceptional,  Very  Good,  Satisfactory,  Marginal,  and 
Unsatisfactory.  It  should  be  noted  that  the  contractor  is  allowed  to  review  the  CPAR 
assessment  and  provide  comments  back  to  the  government  assessing  official  prior 
to  the  government’s  finalizing  the  CPAR  report. 

During  the  source  selection  phase  of  a  government-negotiated  procurement, 
contractor  performance  information  is  used  in  evaluating  offerors  and  in  making  a 
contract  award  decision  (Rendon  &  Snider,  2008).  In  this  phase,  the  government 
agency  accesses  the  contractor  performance  information  through  the  DoD  Past 
Performance  Information  Retrieval  System  Report  Cards  (PPIRS-RC)  database. 
During  source  selection  in  the  evaluation  of  offeror’s  proposals,  the  government 
agency  uses  the  contractor  past  performance  information  to  determine  whether  the 
offeror  meets  the  required  standards  of  responsibility  as  stated  in  the  federal 
procurement  policy,  and,  depending  on  the  basis  of  award  stipulated  in  the 
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solicitation,  will  use  the  contractor’s  past  performance  ratings  to  justify  an  award  to  a 
higher  priced  offeror. 

The  contractor  performance  information  reported  in  CPARS  and  accessible 
through  PPIRS  provides  outcome-based  data  that  can  be  used  to  identify  successful 
contracts.  The  successful  contracts  determined  by  using  contractor  performance 
information  are  used  in  our  research  methodology  to  identify  the  contract  variables 
that  lead  to  contract  success.  The  research  methodology  for  this  project  is  discussed 
next. 

Research  Design 

With  the  assistance  of  our  MBA  thesis  students  (Hart  et  al.,  2013),  we 
searched  the  PPIRS  database  to  identify  Army  Mission  Installation  Contracting 
Command  (MICC)  services  (non-systems)  contracts  for  the  period  1996-2013.  This 
search  yielded  14,395  contracts  in  total.  The  data  were  then  refined  to  include  only 
those  contracts  associated  with  the  following  product/service  codes: 

•  R:  Professional,  Administrative,  and  Management  Support  Services 

•  J:  Maintenance,  Repair,  and  Rebuilding  of  Equipment  Services 

•  S:  Utilities  and  Housekeeping  Services 

•  D:  Automatic  Data  Processing  and  Telecommunications  Services 

This  data  refinement  yielded  5,621  contracts.  Our  database  was  further  refined  by 
focusing  on  five  Army  MICC  contracting  organizations: 

•  MICC  Region  Fort  Eustis 

•  MICC  Region  Fort  Knox 

•  MICC  Region  Fort  Hood 

•  MICC  Region  Fort  Bragg 

•  MICC  Region  Fort  Sam  Houston 

These  organizations  were  selected  because  they  are  MICC  field  directorate  offices 
(FDOs)  of  this  specific  Army  contracting  command.  This  data  refinement  resulted  in 
715  service  contracts  that  were  used  in  conducting  our  analysis,  as  seen  in  Table  1 . 
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Table  1.  Database  Breakdown 

(Hartet  al„  2013) 


Total  Contracts 

Total  Army  MICC  Non- 
System  Contracts 

14,395 

Less:  Non-R,  J,  S,  D 

Service  Contracts 

8,774 

Total  R,  J,  S,  D  Service 
Contracts 

5,621 

Less:  R,  J,  S,  D  Service 
Contracts  at  other  MICC 

4,906 

R,  J,  S,  D  Service 

Contracts  at  MICC  FDO 
Eustis,  Knox,  Hood, 

Bragg,  Sam  Houston 

715 

Fort  Eustis 

238 

Fort  Knox 

119 

Fort  Hood 

114 

Fort  Bragg 

55 

Fort  Sam  Houston 

189 

For  each  contract,  data  were  collected  on  specific  contract  variables  (type  of 
service,  contract  dollar  value,  level  of  competition,  contract  type)  and  specific 
contractor  assessment  ratings  (quality  of  product/service,  schedule,  cost  control, 
business  relations,  management  of  key  personnel,  and  utilization  of  small  business). 
It  should  be  noted  that  the  data  collected  from  the  PPIRS  database  were  sanitized 
by  removing  identifiable  data  such  as  contract  number,  contractor  name,  DUNS 
number,  and  place  of  performance.  In  addition  to  the  contractor  performance 
information  accessed  from  the  PPIRS-RC  database,  we  also  collected  MICC  region 
organization  demographic  data  (annual  workload  in  dollars,  annual  workload  in 
actions,  number  of  1 102  billets  authorized,  and  percentage  of  1 102  billets  filled  (Hart 
et  al. ,  2013),  These  data  were  also  analyzed  to  determine  whether  these 
organizational  demographics  were  related  to  contract  success. 

Determining  a  contract  to  be  successful  or  unsuccessful  was  made  based  on 
whether  the  contractor  received  a  marginal  or  unsatisfactory  rating  in  any  of  the 
CPAR  assessment  areas  (quality  of  product/service,  schedule,  cost  control, 
business  relations,  management  of  key  personnel,  or  utilization  of  small  business). 
The  contractor’s  receiving  a  marginal  or  unsatisfactory  rating  in  any  one  of  these 
assessment  areas  results  in  the  determination  of  the  contract  as  unsuccessful. 

Based  on  the  analysis  of  the  data  pertaining  to  contract  variables  and 
contractor  assessment  ratings,  we  answered  our  four  primary  research  questions: 


ns. 
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1 .  Do  the  types  of  services  being  acquired  affect  the  success  of  a  service 
contract? 

2.  Do  the  contractual  amounts  affect  the  success  of  a  service  contract? 

3.  Does  the  level  of  competition  used  affect  the  success  of  a  service 
contract? 

4.  Does  the  contract  type  affect  the  success  of  a  services  contract? 

Figure  2  graphically  illustrates  our  research  methodology.  The  column  on  the 
right  contains  the  six  CPARS  assessment  areas.  These  areas  are  used  to  determine 
whether  each  contract  is  successful  or  unsuccessful  based  on  the  CPAR  ratings 
(marginal  or  unsatisfactory).  The  successful  contracts  are  analyzed  using  the  four 
contract  variables  shown  on  the  left  column.  The  purpose  is  to  determine  whether 
there  is  a  relationship  between  contract  variables  and  contract  success. 


Figure  2.  Research  Methodology 

(Hart  etal.,  2013) 

Findings  and  Analysis 

In  this  section,  we  present  an  analysis  of  our  findings.  As  mentioned 
previously,  the  primary  purpose  of  this  research  is  to  determine  if  there  is  a 
relationship  between  contract  variables  (type  of  service,  contract  dollar  value,  level 
of  competition,  contract  type)  and  contract  success  (based  on  contractor 
assessment  rating  of  quality,  schedule,  cost  control,  business  relations, 
management  of  key  personnel,  and  utilization  of  small  business).  With  the 
assistance  of  our  MBA  thesis  students  (Hart  et  al.,  2013),  we  analyzed  the  data  and 
provide  our  findings  at  the  aggregate  level,  as  well  as  the  contract  variable  level. 


Vi  , .  .  1  '  ■■  y  ’f  VJU  d  V 
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Aggregate  Findings 

The  database  consisted  of  715  contracts  accessed  from  the  PPIRS  database 
Of  these  contracts,  22  were  determined  to  be  unsuccessful  based  on  the  CPAR 
assessment  area  ratings,  as  described  in  the  previous  section.  This  resulted  in  a 
total  contract  failure  rate  of  3.08%.  These  results  are  seen  clearly  in  Table  2. 


Table  2.  Total  Contract  Information 

(Hartetal.,  2013) 


Failures 

Success 

Total 

Failure  Rate 

Contracts 

22 

693 

715 

3.08% 

Additionally,  each  assessment  area  was  given  a  score  associated  with  its 
rating  (Exceptional,  5;  Very  Good,  4;  Satisfactory,  3;  Marginal,  2;  Unsatisfactory,  1). 
These  assessment  area  ratings  were  then  averaged  to  examine  what  assessment 
areas  were  rated  higher  throughout  the  population  (see  Table  3).  As  reflected  in 
Table  3,  business  relations  had  the  highest  average  rate  of  failure  among  the  other 
assessment  areas.  The  total  contract  failures  for  each  CPAR  assessment  area  can 
be  seen  in  Figure  3. 


Table  3.  Average  Success  and  Failure  Rates  of  PPIRS  Areas  of  Contract 

Evaluation 

(Hartetal.,  2013) 


Successes 
Average  Ratings 

Failures  Average 
Ratings 

Quality 

4.19 

2.5 

Schedule 

4.19 

2.5 

Cost  Control 

4.1 

2.31 

Business  Relations 

4.17 

3 

Management  of  Key  Personnel 

4.18 

2.68 

Utilization  of  Small  Business 

4.07 

2.5 

- 10 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


Figure  3.  Stated  Reason  of  Failure  Label  for  All  Contracts 

(Hartetal.,  2013) 

The  next  section  discusses  our  research  findings  for  each  contract  variable. 


Type  of  Service 

Finding  1:  The  S  type  services  (Utilities  and  Housekeeping)  had  the  highest 
failure  rate  of  all  the  product  service  codes  analyzed. 

In  answering  the  first  research  question,  “Do  the  types  of  services  being 
acquired  affect  the  success  of  a  service  contract?”,  we  analyzed  the  failure  rates  of 
each  service  type.  The  database  contained  331  R  type  (Professional,  Administrative, 
and  Management  Support)  services,  58  J  type  (Maintenance,  Repair,  and 
Rebuilding  of  Equipment)  services,  292  S  type  (Utilities  and  Housekeeping)  services 
and  34  D  type  (Automatic  Data  Processing  and  Telecommunications)  services  as 
reflected  in  Table  4.  The  failure  rates  are  reflected  in  Figure  4,  and  the  failure 
reasons  are  reflected  in  Figure  5. 

The  S  type  services  had  1 1  contract  failures  resulting  in  a  3.77%  failure  rate. 
Two  reasons  tied  for  the  most  common  reasons  for  S  type  service  failures.  These 
reasons  were  six  business  relation  failures  and  six  failures  due  to  management  of 
key  personnel.  R  type  services  had  nine  labeled  failing  contracts  out  of  331,  giving  R 
type  services  a  failure  rate  of  2.72%  which  was  the  second  lowest.  The  most 
common  reason  for  the  failure  was  quality.  J  type  services  consisted  of  58  contracts 
with  two  labeled  failures.  This  gave  the  J  type  contracts  a  failure  rate  of  3.45%. 
Scheduling  was  listed  as  a  reason  for  both  labeled  failures  of  the  J  type  services. 
There  were  only  34  of  D  service  code  contract  types  with  zero  failures. 
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Table  4.  Type  of  Service  Acquired  Total  Successes  and  Failures 

(Hartetal.,  2013) 


Type  of  Service 
Acquired 
Categories 

Total 

Successes 

Total 

Failures 

Total 

Contracts 

Failure 

Rates 

D 

34 

0 

34 

0.00% 

J 

56 

2 

58 

3.45% 

R 

322 

9 

331 

2.72% 

S 

281 

11 

292 

3.77% 

Total 

693 

22 

715 

3.08% 

4.00% 
3.50% 
3.00% 
£  2.50% 

TO 

cc 

£  2.00% 
LL  1.50% 
1.00% 
0.50% 
0.00% 


2.72% 


R 


3.77% 


3.45% 


J  S 

Type  of  Service  Acquired 


0.00% 

- 1 

D 


Figure  4.  Failure  Rates  of  the  Different  Product  Service  Code  Contracts 

(Hartetal.,  2013) 
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Figure  5.  Reasons  for  Contract  Failure 

(Hartetal.,  2013) 

Although  it  was  found  that  the  type  of  service  was  not  statistically  significant 
in  terms  of  effecting  contract  success,  it  was  observed  that  utilities  and 
housekeeping  service  contracts  failed  six  times  for  business  relations  and  six  times 
for  management  of  key  personnel  out  of  the  1 1  reported  failures.  Mitigating  this 
contract  risk  should  focus  on  the  source  selection  phase,  specifically  the  proposal 
evaluation  activity.  During  this  phase  the  contracting  agency  can  emphasize  its 
assessment  of  the  offeror’s  ability  to  coordinate  its  business  activities  such  as  its 
attitude  towards  customers,  customer  satisfaction,  and  cooperation.  Additionally, 
during  source  selection,  the  contracting  agency  should  increase  its  emphasis  on 
evaluating  the  offeror’s  management  proposal,  and  assessing  the  offeror’s  ability  to 
maintain  qualified  individuals  in  key  positions  as  required  in  the  solicitation. 

Contract  Dollar  Value 

Finding  2:  Contracts  with  a  dollar  value  from  $50  million  to  $1  billion  had  the 
highest  failure  rate  of  all  the  contract  categories. 

In  answering  the  second  research  question,  “Do  the  contractual  amounts 
affect  the  success  of  a  service  contract?”,  we  grouped  the  contracts  into  the 
following  categories:  $0  to  $1  million,  greater  than  $1  million  to  $10  million,  greater 
than  $10  million  to  $50  million,  greater  than  $50  million  to  $1  billion,  and  greater  than 
$1  billion  (see  Table  5).  The  failure  rates  are  reflected  in  Figure  6,  and  the  failure 
reasons  are  reflected  in  Figure  7. 

The  $50  million  to  $1  billion  category  consisted  of  92  contracts  with  eight 
labeled  failures,  giving  it  a  failure  rate  of  8.7%.  This  group’s  most  common  reason 
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for  failing  was  cost  control,  which  was  listed  for  six  failed  contracts.  This  failure  rate 
is  much  higher  than  the  total  contract  average  failure  rate  of  3.08%. 

In  the  first  group  that  consisted  of  contracts  that  were  worth  $0  to  $1  million, 
there  were  a  total  of  35  contracts.  In  this  first  group,  there  was  only  one  labeled  a 
failure.  This  gave  this  group  a  2.86%  failure  rate.  This  contract  was  labeled  a  failure 
because  of  quality. 

The  group  consisting  of  contracts  greater  than  $1  to  $1 0  million  was  the 
largest  of  all  the  grouped  dollar  amounts.  It  consisted  of  466  contracts,  and  of  those 
10  were  labeled  failures.  That  gave  this  group  a  2.15%  failure  rate.  While  this  group 
had  the  most  failures  numerically,  it  still  was  under  the  average  failure  rate  because 
of  the  number  of  contracts  in  this  group.  The  most  common  reason  for  this  group  to 
fail  was  for  quality,  which  was  cited  seven  times. 

The  contracts  in  the  greater  than  $10  to  $50  million  group  consisted  of  1 18 
contracts.  There  were  three  labeled  failures  in  this  group.  This  group  had  a  2.54% 
failure  rate.  This  group  was  also  under  the  average  total  contract  failure  rate  of 
3.08%.  This  contract  group  most  commonly  failed  for  scheduling  issues  and 
management  of  key  personnel.  It  failed  for  scheduling  twice  and  management  of  key 
personnel  twice.  This  means  that  one  of  the  contracts  in  this  group  had  both  issues 
listed  as  reasons  for  failure. 

The  group  consisting  of  contracts  worth  greater  than  $1  billion  was  the 
smallest  group  in  our  contractual  amount  grouping.  It  only  consisted  of  four  contracts 
and  did  not  contain  any  labeled  failures. 


Table  5.  Contract  Amount  Total  Successes  and  Failures 

(Hartetal.,  2013) 


Contract  Dollar 
Amount 
Categories 

Total 

Successes 

Total 

Failures 

Total 

Contracts 

Failure 

Rates 

$0-$1 M 

34 

1 

35 

2.86% 

>$1M-$10M 

456 

10 

466 

2.15% 

>$10M-$50M 

115 

3 

118 

2.54% 

>$50M-$1  B 

84 

8 

92 

8.70% 

>$1  B 

4 

0 

4 

0.00% 

Total 

693 

22 

715 

3.08% 
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Figure  6.  Failure  Rate  by  Grouped  Dollar  Value 

(Hartetal.,  2013) 
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Figure  7.  Reasons  for  Grouped  Dollar  Value  Failure 

(Hartetal.,  2013) 

Our  analysis  found  that  the  contract  dollar  value  does  have  a  significant 
impact  on  service  contracting  success.  Contracts  with  dollar  values  in  the  $50 
million— $1  billion  range  had  the  highest  failure  rate  at  8.7%.  The  most  common 
reason  for  this  failure  was  cost  control.  Assuming  that  the  awarded  contract  was 
proper  (based  on  an  accurate  government  cost  estimate),  mitigating  this  contract 
risk  should  focus  on  the  source  selection  phase.  During  the  evaluation  of  the 
offeror’s  cost  proposal,  the  source  selection  team  should  accurately  assess  the 
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offeror’s  ability  to  forecast,  manage,  and  control  the  cost  associated  with  conducting 
the  services.  Additionally,  the  source  selection  team  must  ensure  that  all  source 
selection  evaluators  are  properly  trained  on  how  to  accurately  evaluate  the  offeror’s 
cost  proposal  to  ensure  the  business  fully  understands  the  contract  requirements. 

Our  analysis  also  found  that  the  group  of  contracts  with  values  ranging  from 
$1  million— $1 0  million  had  a  total  of  10  failures,  with  seven  failures  listed  for  quality 
reasons.  Although  not  statistically  significant,  this  reason  for  failure  deserves  further 
discussion.  Mitigating  the  contract  quality  risk  should  focus  on  the  technical  proposal 
evaluation  during  the  source  selection  phase.  The  source  selection  team  should  be 
thoroughly  knowledgeable  about  the  technical  and  quality  requirements  of  the 
solicitation.  Only  with  a  competent  source  selection  team  can  a  sufficient  evaluation 
of  the  offeror’s  technical  and  quality  approach  can  be  conducted.  A  thorough 
evaluation  of  the  offer’s  technical  proposal  will  reveal  whether  the  offeror  has  the 
technical  capability  to  meet  the  contract  quality  requirements. 

Level  of  Competition 

Finding  3:  Contracts  awarded  competitively  had  the  highest  failure  rate  when 
compared  to  the  other  contracts. 

In  answering  the  third  research  question,  “Does  the  level  of  competition  affect 
the  success  of  a  service  contract?”,  we  grouped  the  contracts  into  three  categories: 
competitive,  non-competitive,  and  other.  The  researchers  looked  at  these  different 
categories  separately  and  examined  the  failure  rate  of  each  group  as  reflected  in 
Table  6.  The  failure  rates  are  reflected  in  Figure  8,  and  the  failure  reasons  are 
reflected  in  Figure  9. 

Of  the  540  competitive  contracts,  17  were  labeled  as  failures,  which  yields  a 
failure  rate  of  3.15%.  The  reasons  that  most  often  resulted  in  a  contract  failure  were 
in  the  areas  of  schedule  and  cost  control,  which  were  each  referenced  seven  times. 
The  next  highest  referenced  source  of  failure  was  management  of  key  personnel, 
which  was  referenced  six  times. 

Non-competitive  contracts  had  the  next  highest  failure  rate  at  2.91%.  There 
were  172  non-competitive  contracts  in  the  database,  of  which  five  were  labeled 
failures.  Quality  was  referenced  four  times  while  schedule,  management  of  key 
personnel,  and  business  relations  were  each  referenced  twice. 

Contracts  competed  as  Other  had  three  contracts  in  the  database  with  zero 
labeled  failures. 
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Table  6.  Level  of  Competition  Total  Successes  and  Failures 

(Hartet  al.,  2013) 


Level  of  Competition 
Categories 

Total 

Successes 

Total 

Failures 

Total 

Contracts 

Failure 

Rates 

Basis  (Competitive) 

523 

17 

540 

3.15% 

Basis  (Non-Competitive) 

167 

5 

172 

2.91% 

Basis  (Other) 

3 

0 

3 

0.00% 

Total 

693 

22 

715 

3.08% 

3.50% 

3.00% 

2.50% 

to 

CD 

+■» 

5  2.00% 

oj 

'5  1.50% 

u. 

1.00% 

0.50% 

0.00% 


3.15% 


Basis  (Competitive)  Basis  (Non-Competitive)  Basis  (Other) 


Basis  of  Contract  Award 


Figure  8.  Failure  Rates  Among  Level  of  Competition 

(Hart  etal.,  2013) 


Figure  9.  Levels  of  Competition  Stated  Reasons  for  Failure 

(Hart  etal.,  2013) 
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Our  research  found  that  competitively  awarded  contracts  had  the  highest  of 
the  category’s  failure  rates  at  3.15%.  These  contracts  failed  seven  times  for 
scheduling  reasons  and  seven  times  for  cost  control.  The  implications  of  these 
findings  point  to  the  proposal  evaluation  aspect  of  the  source  selection  process.  The 
source  selection  evaluators  must  fully  understand  the  cost,  schedule,  and 
performance  requirements  of  the  solicitation  in  order  to  properly  evaluate  the 
offeror’s  proposal.  The  offeror’s  proposal  will  reflect  its  understanding  of  the  contract 
requirements  as  well  as  its  capability  for  meeting  those  requirements.  If  the 
government  does  not  do  an  adequate  job  in  evaluating  the  offeror’s  proposal,  it 
increases  the  risk  for  contractor  schedule  delays  and  cost  overruns. 

Contract  Type 

Finding  4:  Contracts  structured  as  a  combination  contract  had  the  highest 
failure  rate  when  compared  to  the  other  five  types  of  available  contracts. 

In  answering  the  fourth  research  question,  “Does  the  contract  type  affect  the 
success  of  a  service  contract?”,  we  grouped  the  contracts  into  six  categories:  Cost 
Plus  Award  Fee  (CPAF),  Combination,  Cost  Plus  Fixed  Fee  (CPFF),  Cost  Plus 
Incentive  Fee  (CPIF),  Firm  Fixed-Price  (FFP),  and  Other.  The  Other  category 
includes  all  the  contracts  that  did  not  fit  into  the  previous  five  categories,  such  as 
Labor  Hours,  and  Time  and  Materials.  Table  7  presents  the  number  of  failures  for 
each  contract  category.  The  failure  rates  are  reflected  in  Figure  10,  and  the  failure 
reasons  are  reflected  in  Figure  1 1 . 

There  were  four  combination  contracts  examined  in  the  database.  Of  these 
four  contracts,  two  were  labeled  failures,  which  yields  a  failure  rate  of  50.0%. 
Schedule  and  cost  were  both  referenced  twice  in  the  failed  contracts,  while  quality 
and  management  of  key  personnel  were  each  referenced  once. 

Cost  plus  fixed  fee  contracts  had  the  next  highest  failure  rate  at  5.56%.  There 
were  36  CPFF  contracts  in  the  database,  of  which  two  were  labeled  failures.  Cost 
control  was  referenced  twice,  and  schedule  was  referenced  once  in  the  failed 
contracts. 

Contracts  competed  as  cost  plus  award  fee  had  58  contracts  in  the  database, 
with  three  of  them  labeled  as  failures.  This  yielded  a  failure  rate  of  5.17%.  Two  of 
these  failed  contracts  referenced  cost  control  and  business  relations,  while  one 
referenced  the  management  of  key  personnel. 

Firm  fixed-price  contracts  had  524  contracts  in  the  database,  with  14  of  them 
labeled  as  failures.  This  yielded  a  failure  rate  of  2.68%.  Seven  of  these  failed 
contracts  referenced  quality,  while  six  referenced  the  management  of  key  personnel. 
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Other  contract  types  had  89  contracts  in  the  database  with  one  labeled  as  a 
failure  due  to  quality  and  schedule,  which  yielded  a  failure  rate  of  1.12%.  There  were 
four  cost  plus  incentive  fee  contracts,  which  had  zero  labeled  failures. 


Table  7.  Contract  Type  Total  Successes  and  Failures 

(Hartetal.,  2013) 


Contract  Type 
Categories 

Total 

Successes 

Total 

Failures 

Total 

Contracts 

Failure 

Rates 

CPAF 

55 

3 

58 

5.17% 

Combination 

2 

2 

4 

50.00% 

CPFF 

34 

2 

36 

5.56% 

CPIF 

4 

0 

4 

0.00% 

FFP 

510 

14 

524 

2.67% 

Other 

88 

1 

89 

1.12% 

Total 

693 

22 

715 

3.08% 

Figure  10.  Contract  Type  Failure  Rate 

(Hartetal.,  2013) 
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Figure  1 1 .  Contract  Type  Stated  Reasons  for  Failure 

(Hartetal.,  2013) 

The  type  of  contract  used  in  these  services  acquisition  was  statistically 
significant  in  relation  to  contract  success.  Of  three  categories  of  contract  types 
(combination,  CPAF,  and  CPIF),  the  combination  type  showed  a  failure  rate  of  50% 
(two  failures  out  of  four),  CPAF  showed  a  failure  rate  of  5.17%,  and  CPFF  a  failure 
rate  of  5.56%.  These  last  two  categories  are  double  the  failure  rate  of  the  FFP 
category  which  was  2.67%.  The  leading  and  recurring  cause  of  contract  failure  for 
these  contracts  was  cost  control.  The  implication  of  this  finding  focuses  us  on  the 
procurement  planning  phase  of  services  acquisition.  The  government’s  effort  during 
procurement  planning,  specifically  the  requirements  analysis  and  the  market 
research  activities,  should  ensure  that  the  requirement  and  the  market  are 
sufficiently  analyzed  in  order  to  select  the  appropriate  contract  type.  The  more 
defined  the  requirement,  the  more  appropriate  it  is  to  use  a  fixed-price  contract.  In 
the  event  that  a  cost-type  contract  is  selected,  the  government  should  ensure  that  an 
accurate  cost  estimate  is  developed  to  be  used  in  cost  negotiations.  Additionally, 
during  the  source  selection  phase,  specifically  the  cost  proposal  evaluation  activity, 
the  government  should  ensure  that  the  offeror’s  cost  proposal  accurately  reflects  a 
thorough  understanding  of  the  service  effort  and  the  related  costs.  If  the  government 
does  not  do  an  adequate  job  in  evaluating  the  offeror’s  proposal,  it  increases  the  risk 
for  contractor  schedule  delays  and  cost  overruns.  If  the  government  does  not 
conduct  a  sufficient  evaluation  of  the  offeror’s  cost  proposal,  it  may  award  the 
contract  to  an  offeror  who  does  not  have  an  adequate  understanding  of  the  service 
requirement  and  thus  leads  to  greater  possibility  of  cost  overruns.  In  addition  to  the 
contract  variables  previously  discussed,  we  also  analyzed  other  variables  to 
determine  whether  they  had  any  effect  on  contract  success.  The  other  variables 
include  MICC  annual  workload  (by  dollar  value),  MICC  annual  workload  (by  number 
of  actions),  number  of  1 102  billets,  and  percentage  of  1 102  billets  filled.  We  also 
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analyzed  the  data  in  terms  of  workload  (in  dollars)  per  filled  billet.  The  next  section 
discusses  the  findings  related  to  these  variables. 

Significance  Testing 

We  further  analyzed  our  data  to  determine  whether  any  of  the  variables  had  a 
significant  relationship  with  contract  success  by  specifically  looking  at  the  contract 
failure  rates.  We  used  the  chi-square  test  (Fisher’s  exact  test)  to  test  whether  the 
actual  failure  rates  are  significantly  different  than  from  what  would  be  expected  if  the 
total  contract  failure  rate  was  applied  to  each  variable.  The  null  hypothesis  for  this 
test  was  that  the  category  failure  rates  within  the  variables  are  not  significantly 
different  from  the  total  contract  failure  rate  (3.08%).  We  reject  the  null  hypothesis  if 
the  p-value  for  the  variable  is  less  than  .05.  The  results  of  the  chi-square  test  are 
reflected  in  Table  8. 

Table  8.  Chi-Square  and  Fisher’s  Exact  p-Value  Test  Results 

(Hartetal.,  2013) 


Contract  Variables 

p-value 

Significant? 

Type  of  Service  (RJSD) 

0.761 

No 

Contractual  Amounts 

0.036 

Yes 

Level  of  Competition 

1.00 

No 

Contract  Type 

0.009 

Yes 

The  chi-square  test  resulted  in  only  two  variables  being  significantly  different 
when  the  average  total  contract  failure  rate  is  applied  to  the  variable  (3.08%).  These 
variables  were  contract  dollar  value  (p  =  .036)  and  contract  type  (p  =  .009).  In  the 
contract  dollar  value  variable,  the  category  of  $50  million  to  $1  billion  had  an 
expected  failure  rate  of  1  but  had  an  actual  failure  rate  of  8.  In  the  contract  type 
variable,  the  category  of  Combination  Type  contracts  was  expected  to  have  a  .1 
failure  but  actually  had  2  failures.  Additionally,  FFP  Type  contracts  were  expected  to 
have  16  failures  but  actually  had  14  failures.  Finally,  Other  Type  contracts  were 
expected  to  have  3  failures  but  only  had  1  failure. 

MICC  Workload  in  Dollars 

Finding  5:  Regional  MICC  offices  that  had  spent  between  $0  and  $500  million 
in  annual  workload  had  the  highest  failure  rate. 

We  grouped  the  contracts  into  three  categories  based  on  the  MICC  regional 
office’s  annual  workload  (dollars  obligated).  The  categories  consisted  of  total  dollars 
obligated  between  $0  and  $500  million,  greater  than  $500  million  to  $1  billion,  and 
greater  than  $1  billion.  This  is  reflected  in  Table  9.  The  failure  rates  are  reflected  in 
Figure  12,  and  the  failure  reasons  are  reflected  in  Figure  13. 
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The  first  category,  MICC  regions  that  obligated  between  $0  and  $500  million, 
consisted  of  344  contracts  and  had  a  failure  rate  of  4.36%.  The  primary  reason 
stated  for  failure  was  schedule  for  this  category. 

The  second  category,  MICC  regions  that  obligated  greater  than  $500  million 
to  $1  billion,  consisted  of  256  contracts  with  a  failure  rate  of  2.34%.  Management  of 
key  personnel  was  stated  as  the  reason  for  failures. 

The  third  category,  obligations  greater  than  $1  billion,  contained  1 15  contracts 
with  only  one  of  them  being  labeled  a  failure  giving  it  a  .87%  failure  rate.  The  labeled 
failure  listed  cost  control  and  business  relations  as  the  reason  for  failure. 

Table  9.  Annual  Workload  in  Dollars  Total  Success  and  Failures 

(Hartetal.,  2013) 


Contract  Dollar 
Amount  Categories 

Total 

Successes 

Total 

Failures 

Total 

Contracts 

Failure 

Rates 

$0-$500M 

329 

15 

344 

4.36% 

>$500M-$1  B 

250 

6 

256 

2.34% 

>$1  B 

114 

1 

115 

0.87% 

Total 

693 

22 

715 

3.08% 

Figure  12.  Annual  Workload  in  Dollars  Failure  Rates 

(Hartetal.,  2013) 
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1  Quality 

<  Schedule 

1  Cost  Control 

1  Business  Relations 

Management  of  Key 
Personnel 

1  Utilization  of  Small 
Business 


Figure  13.  Annual  Workload  in  Dollars  Stated  Reason  for  Failures  Among 

the  Ratings 

(Hartetal.,  2013) 


MICC  Workload  in  Actions 


Finding  6:  MICCs  that  completed  3,501  to  7,000  contract  actions  annually  had 
the  highest  failure  rate  when  compared  to  MICCs  that  completed  3,500  or 
fewer  contract  actions. 


We  grouped  the  contracts  into  two  categories  based  on  the  MICC  regional 
office’s  annual  workload  (actions  completed).  The  first  category  ranged  from  0  to 
3,500  contract  actions.  The  second  category  ranged  from  3,501  to  7,000  contract 
actions.  This  is  reflected  in  Table  10.  The  failure  rates  are  reflected  in  Figure  14,  and 
the  failure  reasons  are  reflected  in  Figure  15. 

The  3,501  to  7,000  contract  actions  category  consisted  of  277  contracts.  Of 
these  277  contracts,  nine  were  labeled  as  failures,  which  yielded  a  failure  rate  of 
3.25%.  Quality  and  the  management  of  key  personnel  were  each  referenced  five 
times  in  the  failed  contracts. 


The  0  to  3,500  contract  actions  category  consisted  of  413  contracts  of  which 
12  were  labeled  as  failures,  resulting  in  a  failure  rate  at  2.91%.  Schedule  and  cost 
control  were  each  referenced  five  times  while  quality  was  referenced  four  times  in 
the  failed  contracts.  There  were  no  data  available  for  the  25  contracts  of  which  one 
was  labeled  a  failure. 
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Table  10.  Number  of  Contract  Actions  Versus  Total  Success  and  Failures 

(Hartetal.,  2013) 


Number  of 
Actions 
Categories 

Total 

Successes 

Total 

Failures 

Total 

Contracts 

Failure 

Rates 

0-3,500 

401 

12 

413 

2.91% 

3,501-7,000 

268 

9 

277 

3.25% 

Data  not  available 

24 

1 

25 

4.00% 

Total 

693 

22 

715 

3.08% 

3.30% 

3.20% 

in 
Q) 

+■» 
ru 
CC 
CD 

'ro 

LL. 

2.80% 

2.70% 


3.10% 

3.00% 

2.90% 


2.91% 


0-3500 


3.25% 


3501-7000 


Number  of  Completed  Contracts  Annually 


Figure  14.  Failure  Rates  by  Number  of  Completed  Contracts  Annually 

(Hartetal.,  2013) 
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Figure  1 5.  Number  of  Completed  Contracts  Annually  Stated 

Reasons  for  Failure 

(Hartetal.,  2013) 

1102  Billets  (Authorized) 

Finding  7:  The  category  with  0  to  50  1102  billets  had  the  highest  failure  rate. 

We  grouped  the  contracts  into  three  categories  based  on  the  number  of  1 102 
billets  authorized  for  each  MICC  regional  office.  The  first  category  consisted  of  MICC 
regional  offices  that  had  0  to  50  1102  billets,  the  second  category  consisted  of  MICC 
regional  offices  that  had  over  50  authorized  1 102  billets.  (There  were  25  contracts 
that  did  not  have  any  data  available).  This  is  reflected  in  Table  1 1 .  The  failure  rates 
are  reflected  in  Figure  16,  and  the  failure  reasons  are  reflected  in  Figure  17. 

MICC  regional  offices  that  had  0  to  50  authorized  1102  billets  consisted  of 
147  contracts  with  eight  failures,  giving  these  MICC  offices  a  failure  rate  of  5.44%. 
The  common  reason  for  these  failures  was  quality.  This  reason  was  listed  for  five  of 
the  eight  labeled  failed  contracts. 

MICC  regional  offices  that  had  over  50  authorized  1102  billets  consisted  of 
543  contracts  with  13  failures,  giving  this  group  a  2.39%  failure  rate.  The  most 
common  reason  for  failure  listed  for  this  group  was  schedule.  This  reason  was  listed 
seven  times  out  of  the  13  labeled  failed  contracts. 

The  final  category  did  not  have  data  on  authorized  billets.  This  group 
contained  25  total  contracts  with  one  failure,  giving  this  category  a  4%  failure  rate. 
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Table  11.  Authorized  1102  Billets  Total  Success  and  Failures 

(Hart  et  al. ,  2013) 


Authorized  1102 
Billets  Categories 

Total 

Successes 

Total 

Failures 

Total 

Contracts 

Failure 

Rates 

0-50 

139 

8 

147 

5.44% 

51-105 

530 

13 

543 

2.39% 

Data  not  available 

24 

1 

25 

4.00% 

Total 

693 

22 

715 

3.08% 

6.00% 

5.00% 

„  4.00% 

a ) 

+-» 

to 

qj  3.00% 
™  2.00% 
1.00% 

0.00% 


5.44% 


0-50 

Total  Number  of  Authorized  1102  Billets 


2.39% 


51-105 


Figure  16.  Failure  Rates  of  MICCs  by  1102  Billet  Authorizations 

(Hart  et  al.,  2013) 
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Figure  17.  Reasons  for  Failures  by  Billet  Authorization 

(Hartetal.,  2013) 

1102  Billets  (Percentage  Filled) 

Finding  8:  As  the  percentage  of  1102  filled  billets  increased,  the  contract 
failure  rate  decreased. 

We  grouped  the  contracts  into  six  categories  based  on  the  percentage  of 
1102  billets  filled  for  each  MICC  regional  office.  The  categories  consisted  of  MICC 
regional  offices  that  had  billets  50-60%  filled,  61-70%  filled,  71-80%  filled,  81-90% 
filled,  and  91-100%  filled.  (There  were  also  contract  data  that  did  not  contain 
adequate  billet  information.)  This  finding  is  reflected  in  Table  12.  The  failure  rates 
are  reflected  in  Figure  18  and  the  failure  reasons  are  reflected  in  Figure  19. 

The  MICC  region  group  of  1 102  billets  that  were  61-70%  filled  had  81 
contracts,  with  a  4.94%  failure  rate.  The  most  common  reasons  listed  for  this  group 
to  fail  were  schedule  and  cost  control. 

For  the  MICC  region  group  of  1 102  billets  that  were  50-60%  filled  had  22 
contracts  with  one  labeled  failure,  which  gave  this  group  a  4.55%  failure  rate.  This 
rate  is  higher  than  the  total  contract  average  of  3.08%.  The  reason  for  the  failure  in 
this  group  was  quality,  scheduling,  and  management  of  key  personnel. 

The  next  group  of  1 102  billets  that  were  71-80%  filled  consisted  of  122  total 
contracts  with  five  contracts  labeled  as  failures,  giving  this  group  a  4.1%  failure  rate. 
This  group  was  higher  than  the  total  contract  failure  rate  of  3.08%.  The  most 
frequent  reasons  listed  for  the  contract  failures  were  schedule  and  cost  control. 
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The  group  that  contained  billets  filled  81-89%  had  233  contracts  with  five 
labeled  as  failures,  giving  this  group  a  2.15%  failure  rate.  The  most  common 
reasons  for  failure  of  these  contracts  were  quality,  schedule,  and  business  relations. 

The  data  for  the  final  group  that  had  billets  90-100%  filled  consisted  of  99 
total  contracts.  It  contained  one  labeled  failure,  giving  this  group  a  failure  rate  of 
1 .01  %.  The  reason  for  the  failure  in  this  group  was  management  of  key  personnel. 

Table  12.  Percentage  of  1102  Billets  Filled  Total  Success  and  Failures 

(Hart  etal.,  2013) 


Percentage  of  1102 
Billet  Filled 
Categories 

Total 

Successes 

Total 

Failures 

Total 

Contracts 

Failure 

Rates 

50-60% 

21 

1 

22 

4.55% 

61-70% 

77 

4 

81 

4.94% 

71-80% 

117 

5 

122 

4.10% 

81-89% 

228 

5 

233 

2.15% 

90-100% 

98 

1 

99 

1.01% 

Data  not  available 

152 

6 

158 

0.00% 

Total 

693 

22 

715 

3.08% 

Figure  18.  Failure  Rate  for  Different  1102  Billet  Vacancy  Groups 

(Hart  etal.,  2013) 
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Figure  19.  Reasons  for  Failures  of  Different  1102  Vacancy  Groups 

(Hart  etal.,  2013) 


Although  it  was  found  that  the  percentage  of  1 102  billets  filled  was  not 
statistically  significant  in  terms  of  effecting  contract  success,  it  was  observed  that  as 
the  percentage  of  1 102  filled  billets  increased,  the  contract  failure  rate  decreased. 
Thus,  as  the  MICCs  contracting  workforce  increased,  the  contract  success  rate 
increased.  This  finding  supports  the  previous  reports  on  the  importance  of  having 
trained  1102  personnel  (GAO,  2002b,  2009a;  Gansler,  2011).  Having  a  sufficient 
contracting  workforce  enables  the  government  to  perform  due  diligence  not  only  in 
the  source  selection  phase  of  the  contract,  but  also  in  the  contract  administration 
phase  as  well. 

Workload  Per  Filled  Billet 

Finding  9:  As  workload  dollars  per  filled  billet  increases,  contractor 
performance  ratings  also  increase,  and  thus  contract  failure  ratings  decrease. 

In  addition  to  analyzing  the  data  in  terms  of  workload  (both  dollars  and 
actions)  as  well  as  billets  (both  authorized  and  percentage  filled),  we  also  analyzed 
the  data  by  looking  at  workload  (in  dollars)  per  filled  billet  as  it  correlated  with 
associated  failure  rates  and  average  CPARS  ratings  across  seven  dimensions  for 
which  contracts  are  rated.  Our  purpose  here  was  to  determine  whether  workload  (in 
dollars)  per  filled  billet,  as  a  measure  of  contracting  work  being  performed  by  an 
individual,  is  related  to  contract  failure  rate,  as  reflected  in  performance  outcome. 
Table  13  reflects  the  workload  (in  dollars)  per  filled  billet,  for  each  MICC,  for  each 
year.  Also  reflected  in  the  table  is  the  average  rating  for  each  performance 
evaluation  area,  as  well  as  the  contract  failure  rate. 
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We  determined  a  coefficient  of  correlation  between  each  average 
performance  and  workload  per  filled  billet.  A  positive  correlation  indicates  that  both 
workload  and  the  performance  measure  increase  together.  In  addition,  we  created 
scatter  plots  (see  Figure  20)  to  visual  analyze  the  relationship  between  workload  per 
filled  billet  and  each  of  the  individual  performance  measures.  The  results  from  Fort 
Eustis  in  2007  were  excluded  from  both  the  correlation  determination  and  the  scatter 
plots  since  the  workload  per  filled  billet  of  $45  million  was  determined  to  be  an 
outlier. 

As  reflected  in  the  table  and  charts,  the  correlations,  although  not  very  strong, 
indicate  that  as  workload  dollars  per  filled  billet  increase,  contractor  performance 
ratings  also  increase,  and  thus  contract  failure  ratings  decrease.  Although  the 
correlations  are  not  particularly  strong,  they  do  provide  some  interesting  preliminary 
findings  that  deserve  further  exploration.  This  relationship  could  be  explained  by 
reasoning  that  the  MICC  with  higher  workload  (dollars  per  filled  billets)  may  not  have 
the  resources  to  ensure  due  diligence  in  conducting  thorough  contractor 
performance  assessments  and  thus  may  be  inflating  the  performance  ratings. 
Another  explanation  may  be  that  MICCS  with  higher  workloads  may  have  more 
experienced  and  competent  personnel  and  more  capable  contracting  processes  that 
are  resulting  in  better  contracts  and  successful  contractors.  As  previously  stated, 
although  these  correlations  are  not  particularly  strong,  they  do  provide  some 
interesting  preliminary  findings  that  deserve  further  exploration  in  future  research. 

The  next  section  summarizes  our  research,  presents  our  conclusions,  and 
identifies  our  recommendations. 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


■30- 


Table  13.  Workload  (in  Dollars)  Per  Filled  Billet,  Performance  Ratings,  and  Failure  Rates 


M  ICC/Year 

Workload($)  by 

Filled  Billet 

Quality 

Schedule 

Cost 

Control 

Business 

Relations 

Management 
of  Key 
Personnel 

Utilization 
of  Small 
Businesses 

Failure 

Rate 

Fort  Sam  2008 

$1,728,074 

4.08 

4.13 

4.09 

4.08 

4.08 

3.85 

0.04 

Fort  Bragg  2012 

$2,967,756 

3.00 

3.00 

3.00 

3.50 

3.83 

0.00 

Fort  Sam  2009 

$3,390,218 

4.07 

3.93 

3.97 

3.98 

3.98 

4.11 

0.05 

Fort  Sam  2010 

$3,390,218 

3.79 

3.78 

3.63 

3.83 

3.75 

3.84 

0.04 

Fort  Sam  201 1 

$3,390,218 

3.88 

3.82 

3.90 

3.88 

3.81 

3.84 

0.04 

Fort  Sam  2012 

$3,390,218 

4.13 

4.00 

3.25 

4.00 

4.14 

4.00 

0.13 

Fort  Hood  2012 

$5,163,911 

3.80 

3.60 

4.00 

3.20 

3.40 

3.00 

0.00 

Fort  Eustis  2012 

$5,768,758 

4.55 

4.30 

4.19 

4.37 

4.28 

4.83 

0.00 

Fort  Eustis  201 1 

$6,790,396 

4.33 

4.28 

4.07 

4.41 

4.32 

4.06 

0.01 

Fort  Bragg  2011 

$6,871,488 

4.38 

4.25 

4.50 

4.33 

4.38 

5.00 

0.00 

Fort  Hood  2011 

$7,630,889 

3.90 

4.00 

3.77 

3.90 

3.97 

3.71 

0.06 

Fort  Knox  2010 

$8,292,513 

4.38 

4.29 

4.20 

4.33 

4.24 

4.00 

0.05 

Fort  Knox  2011 

$8,292,513 

4.06 

4.09 

4.00 

4.09 

4.03 

3.50 

0.00 

Fort  Hood  2010 

$8,696,449 

3.88 

4.03 

3.81 

3.88 

3.76 

3.92 

0.06 

Fort  Eustis  2010 

$10,528,404 

4.36 

4.43 

4.48 

4.18 

4.58 

4.41 

0.00 

Fort  Hood  2008 

$12,001,524 

4.17 

4.25 

4.09 

4.33 

4.17 

3.50 

0.08 

Fort  Hood  2009 

$12,001,524 

4.22 

4.56 

4.18 

4.28 

4.11 

4.33 

0.06 

Fort  Eustis  2008 

$13,065,094 

4.76 

4.65 

4.56 

4.76 

4.76 

4.20 

0.00 

Fort  Eustis  2009 

$13,065,094 

4.24 

4.27 

4.38 

4.27 

4.48 

4.30 

0.00 

Fort  Knox  2008 

$13,162,890 

4.30 

4.30 

4.00 

4.25 

4.20 

3.50 

0.05 

Fort  Knox  2009 

$13,162,890 

4.23 

4.32 

4.00 

4.52 

4.25 

3.67 

0.08 

Fort  Bragg  2007 

$14,976,974 

3.40 

3.80 

3.67 

4.00 

4.25 

0.20 

Fort  Bragg  2008 

$16,586,239 

4.42 

4.33 

4.22 

4.25 

4.42 

4.40 

0.00 

Fort  Bragg  2009 

$16,586,239 

3.92 

4.00 

3.57 

3.91 

3.70 

0.00 

Fort  Eustis  2007 

$45,646,866 

4.00 

4.10 

4.11 

4.20 

4.20 

4.00 

0.00 

Correlations 

0.277 

0.259 

0.225 

0.281 

0.261 

0.0434 

-0.094 
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D.  Workload  versus  Cost  Control 


E.  Workload  versus  Business  Relations 


F.  Workload  versus  Small  Business 


Figure  20.  Workload  Dollars  Per  Filled  Billet  Versus  Individual  Performance  Measures 
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Summary,  Conclusions,  and  Recommendations 

Summary 

Although  the  service  sector  represents  the  largest  and  the  fastest-growing 
segment  of  the  U.S.  economy  and  comprises  over  half  of  the  DoD’s  procurement 
dollars,  poor  management  practices  have  undermined  the  government’s  ability  to 
obtain  best  value  in  services  acquisition.  The  DoD  must  give  greater  attention  to 
proper  procurement  planning,  source  selection,  and  contract  administration  in 
acquiring  services.  The  use  of  contractor  performance  information  is  a  critical  facet 
of  services  acquisition.  During  the  source  selection  phase  of  the  contract 
management  process,  specifically  proposal  evaluation,  the  DoD  relies  on  the  Past 
Performance  Information  Retrieval  System  (PPIRS)  to  obtain  an  offeror’s  past 
performance  information,  a  required  evaluation  factor  in  source  selections. 
Additionally,  during  the  contract  closeout  phase,  specifically  after  the  contract  period 
of  performance,  the  DoD  assesses  the  contractor’s  performance  and  reports  this 
assessment  in  the  Contractor  Performance  Assessment  Reporting  System 
(CPARS). 

The  objective  of  this  research  was  to  identify  variables  in  the  services 
contracting  process  that  drive  the  success  of  services  acquisition.  Our  research 
approach  included  analyzing  contractor  past  performance  data  obtained  from  the 
DoD  PPIRS  database.  We  accessed  the  PPIRS  database  to  collect  contractor 
performance  ratings  on  completed  services  contracts  to  determine  whether  the 
contracts  are  successful  or  not  successful.  Using  statistical  analysis  to  draw 
conclusions  on  whether  certain  contracting  variables,  type  of  service,  contract  dollar 
value,  level  of  competition,  and  contract  type  affected  the  success  of  the  contract, 
we  generalized  our  research  findings  and  provided  recommendations  for  improving 
the  Army’s  as  well  as  DoD’s  services  acquisition  management.  Our  focus  is  to 
identify  where  the  contract  failures  could  be  addressed,  thus  assisting  DoD 
contracting  officials.  In  presenting  the  conclusion  of  our  research,  we  have  to  first 
assume  that  the  awarded  contract  was  proper  (based  on  a  fair  and  reasonable  price 
and  accurately  reflecting  schedule  and  performance  requirements)  in  terms  of  the 
requiring  agency’s  needs.  With  a  properly  awarded  contract,  the  DoD  can  then 
perform  the  required  contract  administration  activities  in  order  to  identify  deficiencies 
with  the  contractor’s  work  and  ensure  proper  documentation  into  CPARS. 

Conclusion 

The  following  presents  our  research  questions  and  related  findings: 

1 .  Do  the  types  of  services  being  acquired  affect  the  success  of  a  service 
contract? 
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The  S  type  services  (Utilities  and  Housekeeping)  had  the  highest 
failure  rate  of  all  the  product  service  codes  analyzed.  The  S  type 
services  had  1 1  contract  failures,  resulting  in  a  3.77%  failure  rate.  The 
reasons  for  contract  failure  included  business  relations  and 
management  of  key  personnel. 

2.  Do  the  contractual  amounts  affect  the  success  of  a  service  contract? 
Contracts  with  a  dollar  value  from  $50  million  to  $1  billion  had  the 
highest  failure  rate  of  all  the  contract  categories.  The  contracts  with 
dollar  values  between  $50  million  to  $1  billion  category  consisted  of  92 
contracts,  with  eight  labeled  failures  giving  it  a  failure  rate  of  8.7%.  This 
group’s  most  common  reason  for  failing  was  cost  control. 

3.  Does  the  level  of  competition  used  affect  the  success  of  a  service 
contract? 

Contracts  awarded  competitively  had  the  highest  failure  rate  when 
compared  to  the  other  two  forms  of  competition  available.  Of  the  540 
competitive  contracts,  17  were  labeled  as  failures,  which  yield  a  failure 
rate  of  3.15%.  The  reasons  that  most  often  resulted  in  a  contract 
failure  were  in  the  areas  of  schedule  and  cost  control. 

4.  Does  the  contract  type  affect  the  success  of  a  services  contract? 
Contracts  structured  as  a  combination  contract  had  the  highest  failure 
rate  when  compared  to  the  other  five  types  of  available  contracts. 

There  were  four  Combination  contracts  examined  in  the  database.  Of 
these  four  contracts,  two  were  labeled  failures,  which  yields  a  failure 
rate  of  50.0%.  Schedule  and  cost  were  both  referenced  twice  in  the 
failed  contracts,  while  quality  and  management  of  key  personnel  were 
each  referenced  once. 

We  further  analyzed  our  data  to  determine  whether  any  of  the  variables  had  a 
significant  relationship  with  contract  success  by  specifically  looking  at  the  contract 
failure  rates.  We  used  the  chi-square  test  (Fisher’s  exact  test)  to  test  if  the  actual 
failure  rates  are  significantly  different  than  what  would  be  expected  if  the  total 
contract  failure  rate  was  applied  to  each  variable.  The  results  of  the  chi-square  test 
identified  that  Contractual  Amounts  and  Contract  Type  were  our  only  statistically 
significant  variables. 

Finally,  we  looked  at  the  relationships  between  percentage  of  filled  1102 
billets  and  failure  rates,  and  between  workload  dollars  per  filled  billet  and  failure 
rates  and  made  some  interesting  observations.  We  saw  that  as  the  percentage  of 
1 102  filled  billets  increased,  the  contract  failure  rate  decreased.  This  would  seem 
intuitive,  that  as  the  workforce  increases,  the  contract  success  rate  would  also 
increase,  since  there  would  be  sufficient  resources  to  manage  the  contracting 
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process.  However,  we  also  observed  that  as  workload  dollars  per  filled  billet 
increased,  contractor  performance  ratings  also  increased,  and  thus  contract  failure 
ratings  decreased.  This  relationship  appears  counterintuitive  since  any  additional 
workload  in  the  organization  would  place  a  higher  demand  on  the  workforce  thus 
resulting  in  fewer  resources  to  manage  the  contracting  process.  These  two 
observations  need  further  exploration  using  a  much  expanded  contractor 
performance  database. 

Recommendations 

The  overarching  goal  in  our  services  acquisition  research  stream  is  to  identify 
the  drivers  of  acquisition  practices  and  their  relationship  to  successful  service 
contracts  or,  in  other  words,  “What  drives  successful  services  contracts?”  Our 
previous  research  indicated  that  the  DoD  defines  and  measures  a  successful 
services  contract  by  analyzing  the  contract  outcomes  (performance,  cost,  and 
schedule).  We  used  contractor  performance  as  a  surrogate  measure  for  contract 
outcomes;  that  is,  a  successful  contractor  as  reflected  in  the  CPAR  (in  terms  of 
quality,  schedule,  cost  control,  business  relations,  management  of  key  personnel, 
and  utilization  of  small  business)  would  indicate  a  successful  contract.  Thus,  our 
research  used  contractor  performance  ratings  to  determine  if  there  was  any 
relationship  between  contract  variables  and  contract  success.  Based  on  our 
research,  we  provide  the  following  recommendations  for  our  more  meaningful 
research  findings. 

Finding  1  stated  that  the  S  type  services  (Utilities  and  Housekeeping)  had  the 
highest  failure  rate  of  all  of  the  product  service  codes  analyzed.  Although  this  finding 
was  not  statistically  significant,  it  may  indicate  that  contracting  for  utilities  and 
housekeeping  services  may  be  more  difficult  than  for  the  other  service  types. 
Perhaps  the  contracting  agencies  may  need  to  place  additional  emphasis  on 
procurement  planning  (developing  the  utilities  and  housekeeping  services 
performance  work  statement  [PWS])  as  well  as  source  selection  (evaluation  of 
offeror  technical  proposals).  Additionally,  increased  emphasis  during  contract 
administration  (contractor  oversight  and  surveillance)  may  also  improve  the  success 
of  utilities  and  housekeeping  services  contracts. 

Finding  2  shows  that  contracts  with  a  dollar  value  from  $50  million  to  $1  billion 
had  the  highest  failure  rate  of  all  contract  dollar  value  categories.  The  most  common 
reason  for  these  contract  failures  was  due  to  cost  control  factors.  This  finding  was 
found  to  be  statistically  significant  and  provides  an  indication  that  perhaps  contracts 
in  this  dollar  value  range  require  additional  management  review  and  oversight  during 
the  procurement  planning  and  source  selection  phases.  The  contracting  agencies 
may  not  be  providing  adequate  review  for  these  higher  value  contracts  in  the  areas 
of  developing  PWSs,  cost  estimates,  budgets,  and  service  delivery  schedules. 
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Finding  3  identified  that  contracts  awarded  competitively  had  the  highest 
failure  rate  when  compared  to  contracts  awarded  non-competitively.  The  most 
common  reason  for  these  contracts  failures  was  due  to  schedule  and  cost  control 
factors.  Although  this  finding  was  not  statistically  significant,  it  may  indicate  that 
additional  emphasis  may  be  needed  in  conducting  competitive  source  selections. 
Perhaps  the  contracting  agencies  may  need  additional  training  in  offeror  proposal 
evaluation,  specifically  evaluating  cost  and  technical  proposals,  as  well  as  evaluating 
project  schedules.  The  offeror’s  cost  and  technical  proposals  and  schedule  reflects 
the  offeror’s  understanding  of  the  contract  requirements.  If  these  proposals  and 
schedules  are  not  properly  evaluated  by  trained  source  selection  team  members,  the 
contractor  may  experience  cost  overruns  and  schedule  delays. 

Finding  4  identified  that  contracts  structured  as  combination  contracts  had  the 
highest  failure  rate  when  compared  to  the  other  five  types  of  available  contracts 
(CPAF,  CPIF,  CPFF,  FFP,  Other).  This  finding  was  found  to  be  statistically 
significant  and  provides  an  indication  that  perhaps  services  contracts  consisting  of  a 
combination  of  contract  types  are  more  complex  and  require  additional  management 
review  and  oversight  during  the  contracting  process.  The  contracting  agencies  may 
not  be  providing  adequate  review  of  these  more  complex  contracts  in  the  areas  of 
developing  PWSs,  cost  estimates,  budgets,  and  service  delivery  schedules.  In 
addition,  the  contracting  agencies  may  need  to  provide  additional  training  to  the 
contracting  workforce,  specifically  in  the  areas  of  dealing  with  combined  cost  type 
and  fixed  price  services  contracts. 

Our  final  recommendation  is  based  on  Finding  8  which  showed  that  as  the 
percentage  of  1 102  filled  billets  increased,  the  contract  failure  rate  decreased. 
Although  this  was  not  found  to  be  statistically  significant,  it  does  show  that  as  the 
agencies’  contracting  workforce  increased,  the  contract  success  rate  increased  as 
well.  Thus,  our  recommendation  is  that  the  contracting  agencies  strive  to  ensure  that 
there  are  sufficient  1 102  billets  for  each  organization  and  that  these  billets  are 
sufficiently  filled  with  trained  contracting  professionals. 

Our  analysis  of  CPAR  data  identified  some  interesting  areas  worthy  of  further 
exploration.  First,  we  would  like  to  analyze  an  expanded  database  of  services 
contracts,  to  include  other  services  in  addition  to  the  four  service  types  we  initially 
researched.  With  the  expanded  database,  we  can  analyze  overall  contract  failure 
rates,  as  well  as  contract  failure  rates  for  our  selected  contract  variables  (type  of 
service,  dollar  value,  competition  level,  and  contract  type).  Second,  we  would  like  to 
further  explore  the  relationships  between  percentage  of  filled  1 102  billets  and 
contract  failure  rates,  and  between  workload  dollars  per  filled  billet  and  contract 
failure  rates.  Expanding  our  contractor  performance  database  to  include  other 
service  types  will  provide  the  data  integrity  needed  to  identify  any  stronger 
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relationships  among  these  contract  variables.  Finally,  we  would  like  to  analyze  the 
narrative  portion  of  the  contractor  performance  reports  to  determine  alignment  with 
the  objective  performance  ratings,  as  well  as  the  value  added,  not  only  in  the 
narrative  portions,  but  also  in  the  usefulness  of  the  CPARS  as  a  contractor 
assessment  tool. 
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